perf(proxy): one request-scoped Redis pipeline for auth, spend, rate-limit and routing reads by devin-ai-integration[bot] · Pull Request #43407 · BerriAI/litellm

devin-ai-integration · 2026-09-27T03:20:31Z

TLDR

Each request can share a Redis batch across authentication, spend checks, rate limits, and routing reads. Awaiting a result sends queued operations through a pipeline for that Redis backend, and the middleware flushes anything left when the request ends. This PR extends the earlier spend-counter batching work to the rest of the request path

The debug line logged when a routing prefetch fails to arm strips CR and LF from the request's model name and the error text, so a request cannot forge a log entry through it (CodeQL py/log-injection)

Files changed

File	Change
`litellm/caching/redis_batch.py`	Adds the request-scoped queue that sends Redis operations through shared pipelines
`litellm/proxy/auth/auth_object_prefetch.py`	Batches auth-cache reads and write-backs, with the existing direct path as fallback
`litellm/proxy/common_request_processing.py`	Arms routing reads before request admission flushes queued work
`litellm/proxy/hooks/parallel_request_limiter_v3.py`	Sends rate-limit scripts through the shared pipeline and handles failed groups
`litellm/proxy/middleware/redis_request_batch_middleware.py`	Opens a batch for each request and flushes remaining operations at the end
`litellm/proxy/proxy_server.py`	Registers the request batch middleware
`litellm/proxy/spend_tracking/spend_counter_batch.py`	Adds unread spend counters to the request pipeline and collects their results
`litellm/router.py`	Arms cooldown and usage reads for the requested model
`litellm/router_utils/routing_read_batch.py`	Uses prefetched cooldown and usage data during routing, with a fallback read

Request flow

flowchart LR
    R[Request opens batch] --> Q[Auth spend limits and routing queue Redis work]
    Q --> F[Await or request end flushes pending work]
    F --> P[Pipeline per Redis backend sends queued operations]
    P --> C[Callers receive results]

User Flow

Before: a developer whose key has a budget, a team budget, an end-user budget and TPM/RPM limits waits on 15 Redis round trips per request, 9 of them before the provider is called

They send POST https://litellm-domain/v1/chat/completions with "model": "gpt-group" and "user": "perf-enduser"
The proxy reads their identity from Redis, writes it back, reads the spend counters, reserves the budget, runs three separate rate-limit scripts, reads the deployment cooldown state, then checks the response cache: nine waits on Redis, one after the other
The response comes back 200 with the mock reply

After: the same request waits on 5 Redis round trips before the provider is called, gets the same 200 and records the same spend

They send the same POST https://litellm-domain/v1/chat/completions
The proxy reads identity and spend counters in one pipeline, reserves the budget, then sends the identity write-back, the cooldown read and the RPM script in one pipeline, the two TPM scripts in one more, and checks the response cache
The response comes back 200 with the same mock reply

Relevant issues

Stacked on #43369; merge after it. Third of the stacked PRs for one Redis pipeline pre-call and one post-call (#43320, then LIT-8881, LIT-8882, LIT-8883); design and per-request measurements are on LIT-8881

Linear ticket

Resolves LIT-8882

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
The handful of test files covering my change pass locally, e.g. uv run pytest tests/unit/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Same fixture bytes for both arms, PYTHONPATH=<checkout>, real Redis 6.0.16 at 127.0.0.1:6379, real Postgres, mock deployments (litellm_params.mock_response; the change is on the admission path, not a provider path). Every RedisCache method call and every RedisBatch flush is logged with its caller chain by a sitecustomize tracer loaded through PYTHONPATH; the tables list every call between the request marker and the response marker, minus the jobs that fire on timers (_sync_in_memory_spend_with_redis, daily tag spend flush, config prefetch). Calls are round trips: a pipeline or a Lua script counts once; a PIPELINE row names the ops it carried

Fixture: config_full.yaml with two gpt-group deployments (gpt-dep-a, gpt-dep-b, usage-based-routing-v2 routing group) and two shuffle-group deployments (shuffle-dep-a, shuffle-dep-b, top-level simple-shuffle), Redis response cache on, enable_redis_auth_cache: true. Virtual key with max_budget: 1000, tpm_limit: 10000000, rpm_limit: 100000, in team perf-team (same budget and limits), user with max_budget: 1000, end user perf-enduser on a budget of 1000. Requests are sent 12 s apart so the warm pass hits warm caches; the tables are the warm pass

python litellm/proxy/proxy_cli.py --config ~/perf_rt/config_full.yaml --port <port>
curl -s -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" http://127.0.0.1:<port>/v1/chat/completions -d @chat_warm.req.json
# chat_warm.req.json
{"model":"gpt-group","user":"perf-enduser","messages":[{"role":"user","content":"Reply with the single word: pong B"}],"max_tokens":5}
# shuffle_warm.req.json
{"model":"shuffle-group","user":"perf-enduser","messages":[{"role":"user","content":"Reply with the single word: pong B"}],"max_tokens":5}
# chat_stream_warm.req.json
{"model":"gpt-group","user":"perf-enduser","stream":true,"stream_options":{"include_usage":true},"messages":[{"role":"user","content":"Reply with the single word: pong B"}],"max_tokens":5}
# responses_warm.req.json
{"model":"gpt-group","user":"perf-enduser","input":"Reply with the single word: pong B","max_output_tokens":5}
# messages_warm.req.json
{"model":"gpt-group","max_tokens":5,"messages":[{"role":"user","content":"Reply with the single word: pong B"}],"metadata":{"user_id":"perf-enduser"}}

Before (`e8affcb`, the #43369 tip this PR is stacked on)

POST /v1/chat/completions, non-streaming, usage-based routing

HTTP 200, "model": "gpt-group", content mock reply from b
15 Redis round trips on the request path (9 pre-call, 6 post-call). Rows 1 to 9 are nine serial waits on Redis from five different owners

#	phase	RedisCache op	keys	caller
1	pre	`async_batch_get_cache`	`['<team>_<user>', 'team_membership:837848e9-670e-4`	`_fill_from_redis <- prefetch_auth_objects`
2	pre	`async_set_cache_pipeline_with_ttls`	`(('<team>_<user>', {'user_id': '837848e9-670e-4b34`	`_write_back <- _fill_from_db`
3	pre	`async_batch_get_cache`	`['spend:end_user:perf-enduser', 'spend:key:<key-hash>',`	`_fetch <- _load`
4	pre	`async_increment_pipeline`	`[{'key': 'spend:key:<key-hash>', 'increment_value': <cost>`	`_reserve_counters <- _reserve_reservable_counters`
5	pre	`EVALSHA rate-limiter check`	`['{api_key:<key-hash>}:window', '{api_key:ee876d0d5cd318`	`atch_rate_limiter_script <- should_rate_limit`
6	pre	`EVALSHA tpm check-and-increment`	`['{api_key:<key-hash>}:window', '{api_key:ee876d0d5cd318`	`heck_and_increment_by_n <- reserve_tpm_tokens`
7	pre	`EVALSHA tpm check-and-increment`	`['{team:<team>}:window', '{team:<team>}:tokens']`	`heck_and_increment_by_n <- reserve_tpm_tokens`
8	pre	`async_batch_get_cache`	`['deployment:gpt-dep-a:cooldown', 'deployment:gpt-dep-b:cooldown', 'deployment:shuffle-dep-a:cooldown', 'deplo`	`_cooldown_deployments <- async_get_healthy_deployments`
9	pre	`async_get_cache`	`<key-hash>`	`ache <- _retrieve_from_cache`
10	post	`async_set_cache`	`<key-hash>`	`async_set_cache <- async_add_cache`
11	post	`async_batch_get_cache`	`['spend:end_user:perf-enduser', 'spend:key:<key-hash>',`	`_fetch <- _load`
12	post	`async_increment_pipeline`	`[{'key': 'spend:key:<key-hash>', 'increment_value': <cost>`	`ed <- increment_spend_counters`
13	post	`async_increment`	`gpt-dep-b:openai/gpt-4o-mini:tpm:<min>`	`async_increment_cache <- async_log_success_event`
14	post	`async_batch_get_cache`	`['default_user_id:spend', 'tag:User-Agent: curl', 'tag:User-Agent: curl/7.81.0']`	`async_batch_get_cache <- _read_update_cache_values`
15	post	`EVALSHA rate-limiter check`	`['{api_key:<key-hash>}:tokens', '{user:837848e9-670e-4b3`	`_execute_token_increment_script <- async_increment_tokens_with_ttl_preservation`

POST /v1/chat/completions, non-streaming, simple-shuffle

HTTP 200, "model": "shuffle-group", content mock reply from s
15 Redis round trips on the request path (9 pre-call, 6 post-call), the same shape: the cooldown read (row 8) is its own round trip for every routing strategy

#	phase	RedisCache op	keys	caller
1	pre	`async_batch_get_cache`	`['<team>_<user>', 'team_membership:837848e9-670e-4`	`_fill_from_redis <- prefetch_auth_objects`
2	pre	`async_set_cache_pipeline_with_ttls`	`(('<team>_<user>', {'user_id': '837848e9-670e-4b34`	`_write_back <- _fill_from_db`
3	pre	`async_batch_get_cache`	`['spend:end_user:perf-enduser', 'spend:key:<key-hash>',`	`_fetch <- _load`
4	pre	`async_increment_pipeline`	`[{'key': 'spend:key:<key-hash>', 'increment_value': <cost>`	`_reserve_counters <- _reserve_reservable_counters`
5	pre	`EVALSHA rate-limiter check`	`['{api_key:<key-hash>}:window', '{api_key:ee876d0d5cd318`	`atch_rate_limiter_script <- should_rate_limit`
6	pre	`EVALSHA tpm check-and-increment`	`['{api_key:<key-hash>}:window', '{api_key:ee876d0d5cd318`	`heck_and_increment_by_n <- reserve_tpm_tokens`
7	pre	`EVALSHA tpm check-and-increment`	`['{team:<team>}:window', '{team:<team>}:tokens']`	`heck_and_increment_by_n <- reserve_tpm_tokens`
8	pre	`async_batch_get_cache`	`['deployment:gpt-dep-a:cooldown', 'deployment:gpt-dep-b:cooldown', 'deployment:shuffle-dep-a:cooldown', 'deplo`	`_cooldown_deployments <- async_get_healthy_deployments`
9	pre	`async_get_cache`	`<key-hash>`	`ache <- _retrieve_from_cache`
10	post	`async_set_cache`	`<key-hash>`	`async_set_cache <- async_add_cache`
11	post	`async_batch_get_cache`	`['spend:end_user:perf-enduser', 'spend:key:<key-hash>',`	`_fetch <- _load`
12	post	`async_increment_pipeline`	`[{'key': 'spend:key:<key-hash>', 'increment_value': <cost>`	`ed <- increment_spend_counters`
13	post	`async_increment`	`shuffle-dep-a:openai/gpt-4o-mini:tpm:<min>`	`async_increment_cache <- async_log_success_event`
14	post	`async_batch_get_cache`	`['default_user_id:spend', 'tag:User-Agent: curl', 'tag:User-Agent: curl/7.81.0']`	`async_batch_get_cache <- _read_update_cache_values`
15	post	`EVALSHA rate-limiter check`	`['{api_key:<key-hash>}:tokens', '{user:837848e9-670e-4b3`	`_execute_token_increment_script <- async_increment_tokens_with_ttl_preservation`

POST /v1/chat/completions, streaming

HTTP 200, 8 SSE chunks, "model": "gpt-group", content mock reply from b
16 Redis round trips on the request path (9 pre-call, 7 post-call)

#	phase	RedisCache op	keys	caller
1	pre	`async_batch_get_cache`	`['<team>_<user>', 'team_membership:837848e9-670e-4`	`_fill_from_redis <- prefetch_auth_objects`
2	pre	`async_set_cache_pipeline_with_ttls`	`(('<team>_<user>', {'user_id': '837848e9-670e-4b34`	`_write_back <- _fill_from_db`
3	pre	`async_batch_get_cache`	`['spend:end_user:perf-enduser', 'spend:key:<key-hash>',`	`_fetch <- _load`
4	pre	`async_increment_pipeline`	`[{'key': 'spend:key:<key-hash>', 'increment_value': <cost>`	`_reserve_counters <- _reserve_reservable_counters`
5	pre	`EVALSHA rate-limiter check`	`['{api_key:<key-hash>}:window', '{api_key:ee876d0d5cd318`	`atch_rate_limiter_script <- should_rate_limit`
6	pre	`EVALSHA tpm check-and-increment`	`['{api_key:<key-hash>}:window', '{api_key:ee876d0d5cd318`	`heck_and_increment_by_n <- reserve_tpm_tokens`
7	pre	`EVALSHA tpm check-and-increment`	`['{team:<team>}:window', '{team:<team>}:tokens']`	`heck_and_increment_by_n <- reserve_tpm_tokens`
8	pre	`async_batch_get_cache`	`['deployment:gpt-dep-a:cooldown', 'deployment:gpt-dep-b:cooldown', 'deployment:shuffle-dep-a:cooldown', 'deplo`	`_cooldown_deployments <- async_get_healthy_deployments`
9	pre	`async_get_cache`	`<key-hash>`	`ache <- _retrieve_from_cache`
10	post	`async_set_cache`	`<key-hash>`	`async_set_cache <- async_add_cache`
11	post	`async_set_cache`	`<key-hash>`	`async_set_cache <- async_add_cache`
12	post	`async_batch_get_cache`	`['spend:end_user:perf-enduser', 'spend:key:<key-hash>',`	`_fetch <- _load`
13	post	`async_increment_pipeline`	`[{'key': 'spend:key:<key-hash>', 'increment_value': <cost>`	`s <- _update_database_and_spend_counters_in_batch`
14	post	`async_increment`	`gpt-dep-b:None:tpm:<min>`	`async_increment_cache <- async_log_success_event`
15	post	`async_batch_get_cache`	`['default_user_id:spend', 'tag:User-Agent: curl', 'tag:User-Agent: curl/7.81.0']`	`async_batch_get_cache <- _read_update_cache_values`
16	post	`EVALSHA rate-limiter check`	`['{api_key:<key-hash>}:tokens', '{user:837848e9-670e-4b3`	`_execute_token_increment_script <- async_increment_tokens_with_ttl_preservation`

POST /v1/responses

HTTP 200, "model": "gpt-group", content mock reply from b
18 Redis round trips on the request path (10 pre-call, 8 post-call); the SYNC rows are the pre-existing synchronous cache calls of the Responses path (noted on perf(router): fetch cooldown state and usage counters in one Redis round trip #43320)

#	phase	RedisCache op	keys	caller
1	pre	`async_batch_get_cache`	`['<team>_<user>', 'team_membership:837848e9-670e-4`	`_fill_from_redis <- prefetch_auth_objects`
2	pre	`async_set_cache_pipeline_with_ttls`	`(('<team>_<user>', {'user_id': '837848e9-670e-4b34`	`_write_back <- _fill_from_db`
3	pre	`async_batch_get_cache`	`['spend:end_user:perf-enduser', 'spend:key:<key-hash>',`	`_fetch <- _load`
4	pre	`async_increment_pipeline`	`[{'key': 'spend:key:<key-hash>', 'increment_value': <cost>`	`_reserve_counters <- _reserve_reservable_counters`
5	pre	`EVALSHA rate-limiter check`	`['{api_key:<key-hash>}:window', '{api_key:ee876d0d5cd318`	`_batch_rate_limiter_script <- should_rate_limit`
6	pre	`EVALSHA tpm check-and-increment`	`['{api_key:<key-hash>}:window', '{api_key:ee876d0d5cd318`	`_check_and_increment_by_n <- reserve_tpm_tokens`
7	pre	`EVALSHA tpm check-and-increment`	`['{team:<team>}:window', '{team:<team>}:tokens']`	`_check_and_increment_by_n <- reserve_tpm_tokens`
8	pre	`async_batch_get_cache`	`['deployment:gpt-dep-a:cooldown', 'deployment:gpt-dep-b:cooldown', 'deployment:shuffle-dep-a:cooldown', 'deplo`	`_api_call_with_fallbacks_responses_attempt <- make_call`
9	pre	`async_get_cache`	`<key-hash>`	`_ageneric_api_call_with_fallbacks_responses_attempt <- make_call`
10	pre	`get_cache`	`SYNC <key-hash>`	`get_cache <- _sync_get_cache`
11	post	`set_cache`	`SYNC <key-hash>`	`add_cache <- sync_set_cache`
12	post	`increment_cache`	`SYNC gpt-dep-b:openai/gpt-4o-mini:tpm:<min>`	`increment_cache <- log_success_event`
13	post	`async_set_cache`	`<key-hash>`	`async_set_cache <- async_add_cache`
14	post	`async_batch_get_cache`	`['spend:end_user:perf-enduser', 'spend:key:<key-hash>',`	`_fetch <- _load`
15	post	`async_increment_pipeline`	`[{'key': 'spend:key:<key-hash>', 'increment_value': <cost>`	`ed <- increment_spend_counters`
16	post	`async_increment`	`gpt-dep-b:openai/gpt-4o-mini:tpm:<min>`	`async_increment_cache <- async_log_success_event`
17	post	`async_batch_get_cache`	`['default_user_id:spend', 'tag:User-Agent: curl', 'tag:User-Agent: curl/7.81.0']`	`async_batch_get_cache <- _read_update_cache_values`
18	post	`EVALSHA rate-limiter check`	`['{api_key:<key-hash>}:tokens', '{user:837848e9-670e-4b3`	`_execute_token_increment_script <- async_increment_tokens_with_ttl_preservation`

POST /v1/messages

HTTP 200, "model": "gpt-group", content mock reply from a
31 Redis round trips on the request path (25 pre-call, 6 post-call). Rows 1 to 13 and 17 to 19 are the pre-existing per-object GET/SET/DEL of the Messages path (side finding on LIT-8881, out of scope here); rows 14 to 16 and 20 to 25 are the same admission shape as chat

#	phase	RedisCache op	keys	caller
1	pre	`async_get_cache`	`end_user_id:perf-enduser`	`async_get_cache <- async_get_cache`
2	pre	`async_get_cache`	`end_user_restricted_registry`	`async_get_cache <- async_get_cache`
3	pre	`async_get_cache`	`end_user_restricted_registry`	`async_get_cache <- async_get_cache`
4	pre	`async_set_cache`	`end_user_restricted_registry`	`_cache_registry_answer <- _fetch_and_cache_registry`
5	pre	`async_set_cache`	`end_user_id:perf-enduser`	`async_set_cache <- async_set_cache`
6	pre	`async_get_cache`	`<key-hash>`	`async_get_cache <- async_get_cache`
7	pre	`async_get_cache`	`<key-hash>`	`async_get_cache <- async_get_cache`
8	pre	`async_set_cache`	`<key-hash>`	`async_set_cache <- async_set_cache`
9	pre	`async_get_cache`	`team_id:<team>`	`get_team_object <- _get_hierarchical_router_settings`
10	pre	`async_delete_cache`	`team_id:<team>`	`get_team_object <- _get_hierarchical_router_settings`
11	pre	`async_set_cache`	`team_id:<team>`	`get_team_object <- _get_hierarchical_router_settings`
12	pre	`delete_cache`	`SYNC team_alias:perf-team`	`get_team_object <- _get_hierarchical_router_settings`
13	pre	`async_delete_cache`	`team_alias:perf-team`	`get_team_object <- _get_hierarchical_router_settings`
14	pre	`async_batch_get_cache`	`['<team>_<user>', '837848e9-670e-4b34-936e-156409a`	`_fill_from_redis <- prefetch_auth_objects`
15	pre	`async_set_cache_pipeline_with_ttls`	`(('<user>', {'user_id': '<user>', 'user_alias': No`	`_write_back <- _fill_from_db`
16	pre	`async_batch_get_cache`	`['spend:end_user:perf-enduser', 'spend:key:<key-hash>',`	`_fetch <- _load`
17	pre	`async_get_cache`	`model_access_group_registry`	`cached_registry <- _load_bounded_registry`
18	pre	`async_get_cache`	`model_access_group_registry`	`cached_registry <- _load_bounded_registry`
19	pre	`async_set_cache`	`model_access_group_registry`	`_cache_registry <- _load_bounded_registry`
20	pre	`async_increment_pipeline`	`[{'key': 'spend:key:<key-hash>', 'increment_value': <cost>`	`_reserve_counters <- _reserve_reservable_counters`
21	pre	`EVALSHA rate-limiter check`	`['{api_key:<key-hash>}:window', '{api_key:ee876d0d5cd318`	`h_rate_limiter_script <- should_rate_limit`
22	pre	`EVALSHA tpm check-and-increment`	`['{api_key:<key-hash>}:window', '{api_key:ee876d0d5cd318`	`k_and_increment_by_n <- reserve_tpm_tokens`
23	pre	`EVALSHA tpm check-and-increment`	`['{team:<team>}:window', '{team:<team>}:tokens']`	`k_and_increment_by_n <- reserve_tpm_tokens`
24	pre	`async_batch_get_cache`	`['deployment:gpt-dep-a:cooldown', 'deployment:gpt-dep-b:cooldown', 'deployment:shuffle-dep-a:cooldown', 'deplo`	`pt <- make_call`
25	pre	`async_get_cache`	`<key-hash>`	`ropic_messages_attempt <- make_call`
26	post	`async_set_cache`	`<key-hash>`	`async_add_cache <- _complete_cache_write_despite_cancellation`
27	post	`async_batch_get_cache`	`['spend:end_user:perf-enduser', 'spend:key:<key-hash>',`	`_fetch <- _load`
28	post	`async_increment_pipeline`	`[{'key': 'spend:key:<key-hash>', 'increment_value': <cost>`	`ed <- increment_spend_counters`
29	post	`async_increment`	`gpt-dep-a:None:tpm:<min>`	`async_increment_cache <- async_log_success_event`
30	post	`async_batch_get_cache`	`['default_user_id:spend', 'tag:User-Agent: curl', 'tag:User-Agent: curl/7.81.0']`	`async_batch_get_cache <- _read_update_cache_values`
31	post	`EVALSHA rate-limiter check`	`['{api_key:<key-hash>}:tokens', '{user:837848e9-670e-4b3`	`_execute_token_increment_script <- async_increment_tokens_with_ttl_preservation`

After (`32c0727`; the follow-up commits add suppression reasons, a router coverage exemption and the review fixes: a rejected or failed rate-limit group now refunds every group the pipeline incremented and a Redis denial stands even when another group fails, a reply one op cannot decode fails that op alone, and a cooldown recorded in memory after the prefetch left still wins; the round-trip counts are unchanged)

POST /v1/chat/completions, non-streaming, usage-based routing

HTTP 200, "model": "gpt-group", content mock reply from a (the other deployment of the same group; both served requests in both arms)
11 Redis round trips on the request path (5 pre-call, 6 post-call). Row 1 is the identity MGET and the spend MGET in one pipeline; row 3 is the identity write-back (two SETs), the cooldown read and the RPM rate-limit script in one pipeline; row 4 is the two TPM scripts in one pipeline. Post-call is unchanged (LIT-8883)
Every request of the run returned 200 and the proxy log has no pipeline failure or traceback

#	phase	RedisCache op	keys	caller
1	pre	`PIPELINE MGET+MGET`	`['<team>_<user>', 'team_membership:837848e9-670e-4`	`_read_redis_rows <- _fill_from_redis`
2	pre	`async_increment_pipeline`	`[{'key': 'spend:key:<key-hash>', 'increment_value': <cost>`	`_reserve_counters <- _reserve_reservable_counters`
3	pre	`PIPELINE SET+SET+MGET+EVALSHA`	`['<team>_<user>', 'team_membership:837848e9-670e-4`	`atch_rate_limiter_script <- should_rate_limit`
4	pre	`PIPELINE EVALSHA+EVALSHA`	`['{api_key:<key-hash>}:window', '{api_key:ee876d0d5cd318`	`heck_and_increment_by_n <- reserve_tpm_tokens`
5	pre	`async_get_cache`	`<key-hash>`	`ache <- _retrieve_from_cache`
6	post	`async_set_cache`	`<key-hash>`	`async_set_cache <- async_add_cache`
7	post	`PIPELINE MGET`	`['spend:end_user:perf-enduser', 'spend:key:<key-hash>',`	`_collect_inflight <- _load`
8	post	`async_increment_pipeline`	`[{'key': 'spend:key:<key-hash>', 'increment_value': <cost>`	`ed <- increment_spend_counters`
9	post	`async_increment`	`gpt-dep-a:openai/gpt-4o-mini:tpm:<min>`	`async_increment_cache <- async_log_success_event`
10	post	`async_batch_get_cache`	`['default_user_id:spend', 'tag:User-Agent: curl', 'tag:User-Agent: curl/7.81.0']`	`async_batch_get_cache <- _read_update_cache_values`
11	post	`EVALSHA rate-limiter check`	`['{api_key:<key-hash>}:tokens', '{user:837848e9-670e-4b3`	`_execute_token_increment_script <- async_increment_tokens_with_ttl_preservation`

POST /v1/chat/completions, non-streaming, simple-shuffle

HTTP 200, "model": "shuffle-group", content mock reply from s
11 Redis round trips on the request path (5 pre-call, 6 post-call); the cooldown read now rides the rate limiter's pipeline (row 3) for this strategy too

#	phase	RedisCache op	keys	caller
1	pre	`PIPELINE MGET+MGET`	`['<team>_<user>', 'team_membership:837848e9-670e-4`	`_read_redis_rows <- _fill_from_redis`
2	pre	`async_increment_pipeline`	`[{'key': 'spend:key:<key-hash>', 'increment_value': <cost>`	`_reserve_counters <- _reserve_reservable_counters`
3	pre	`PIPELINE SET+SET+MGET+EVALSHA`	`['<team>_<user>', 'team_membership:837848e9-670e-4`	`atch_rate_limiter_script <- should_rate_limit`
4	pre	`PIPELINE EVALSHA+EVALSHA`	`['{api_key:<key-hash>}:window', '{api_key:ee876d0d5cd318`	`heck_and_increment_by_n <- reserve_tpm_tokens`
5	pre	`async_get_cache`	`<key-hash>`	`ache <- _retrieve_from_cache`
6	post	`async_set_cache`	`<key-hash>`	`async_set_cache <- async_add_cache`
7	post	`PIPELINE MGET`	`['spend:end_user:perf-enduser', 'spend:key:<key-hash>',`	`_collect_inflight <- _load`
8	post	`async_increment_pipeline`	`[{'key': 'spend:key:<key-hash>', 'increment_value': <cost>`	`ed <- increment_spend_counters`
9	post	`async_increment`	`shuffle-dep-a:openai/gpt-4o-mini:tpm:<min>`	`async_increment_cache <- async_log_success_event`
10	post	`async_batch_get_cache`	`['default_user_id:spend', 'tag:User-Agent: curl', 'tag:User-Agent: curl/7.81.0']`	`async_batch_get_cache <- _read_update_cache_values`
11	post	`EVALSHA rate-limiter check`	`['{api_key:<key-hash>}:tokens', '{user:837848e9-670e-4b3`	`_execute_token_increment_script <- async_increment_tokens_with_ttl_preservation`

POST /v1/chat/completions, streaming

HTTP 200, 8 SSE chunks, "model": "gpt-group", content mock reply from a
12 Redis round trips on the request path (5 pre-call, 7 post-call)

#	phase	RedisCache op	keys	caller
1	pre	`PIPELINE MGET+MGET`	`['<team>_<user>', 'team_membership:837848e9-670e-4`	`_read_redis_rows <- _fill_from_redis`
2	pre	`async_increment_pipeline`	`[{'key': 'spend:key:<key-hash>', 'increment_value': <cost>`	`_reserve_counters <- _reserve_reservable_counters`
3	pre	`PIPELINE SET+SET+MGET+EVALSHA`	`['<team>_<user>', 'team_membership:837848e9-670e-4`	`atch_rate_limiter_script <- should_rate_limit`
4	pre	`PIPELINE EVALSHA+EVALSHA`	`['{api_key:<key-hash>}:window', '{api_key:ee876d0d5cd318`	`heck_and_increment_by_n <- reserve_tpm_tokens`
5	pre	`async_get_cache`	`<key-hash>`	`ache <- _retrieve_from_cache`
6	post	`async_set_cache`	`<key-hash>`	`async_set_cache <- async_add_cache`
7	post	`async_set_cache`	`<key-hash>`	`async_set_cache <- async_add_cache`
8	post	`PIPELINE MGET`	`['spend:end_user:perf-enduser', 'spend:key:<key-hash>',`	`_collect_inflight <- _load`
9	post	`async_increment_pipeline`	`[{'key': 'spend:key:<key-hash>', 'increment_value': <cost>`	`s <- _update_database_and_spend_counters_in_batch`
10	post	`async_increment`	`gpt-dep-a:None:tpm:<min>`	`async_increment_cache <- async_log_success_event`
11	post	`async_batch_get_cache`	`['default_user_id:spend', 'tag:User-Agent: curl', 'tag:User-Agent: curl/7.81.0']`	`async_batch_get_cache <- _read_update_cache_values`
12	post	`EVALSHA rate-limiter check`	`['{api_key:<key-hash>}:tokens', '{user:837848e9-670e-4b3`	`_execute_token_increment_script <- async_increment_tokens_with_ttl_preservation`

POST /v1/responses

HTTP 200, "model": "gpt-group", content mock reply from b
14 Redis round trips on the request path (6 pre-call, 8 post-call)

#	phase	RedisCache op	keys	caller
1	pre	`PIPELINE MGET+MGET`	`['<team>_<user>', 'team_membership:837848e9-670e-4`	`_read_redis_rows <- _fill_from_redis`
2	pre	`async_increment_pipeline`	`[{'key': 'spend:key:<key-hash>', 'increment_value': <cost>`	`_reserve_counters <- _reserve_reservable_counters`
3	pre	`PIPELINE SET+SET+MGET+EVALSHA`	`['<team>_<user>', 'team_membership:837848e9-670e-4`	`_batch_rate_limiter_script <- should_rate_limit`
4	pre	`PIPELINE EVALSHA+EVALSHA`	`['{api_key:<key-hash>}:window', '{api_key:ee876d0d5cd318`	`_check_and_increment_by_n <- reserve_tpm_tokens`
5	pre	`async_get_cache`	`<key-hash>`	`_ageneric_api_call_with_fallbacks_responses_attempt <- make_call`
6	pre	`get_cache`	`SYNC <key-hash>`	`get_cache <- _sync_get_cache`
7	post	`set_cache`	`SYNC <key-hash>`	`add_cache <- sync_set_cache`
8	post	`increment_cache`	`SYNC gpt-dep-b:openai/gpt-4o-mini:tpm:<min>`	`increment_cache <- log_success_event`
9	post	`async_set_cache`	`<key-hash>`	`async_set_cache <- async_add_cache`
10	post	`PIPELINE MGET`	`['spend:end_user:perf-enduser', 'spend:key:<key-hash>',`	`_collect_inflight <- _load`
11	post	`async_increment_pipeline`	`[{'key': 'spend:key:<key-hash>', 'increment_value': <cost>`	`ed <- increment_spend_counters`
12	post	`async_increment`	`gpt-dep-b:openai/gpt-4o-mini:tpm:<min>`	`async_increment_cache <- async_log_success_event`
13	post	`async_batch_get_cache`	`['default_user_id:spend', 'tag:User-Agent: curl', 'tag:User-Agent: curl/7.81.0']`	`async_batch_get_cache <- _read_update_cache_values`
14	post	`EVALSHA rate-limiter check`	`['{api_key:<key-hash>}:tokens', '{user:837848e9-670e-4b3`	`_execute_token_increment_script <- async_increment_tokens_with_ttl_preservation`

POST /v1/messages

HTTP 200, "model": "gpt-group", content mock reply from a
27 Redis round trips on the request path (21 pre-call, 6 post-call); the admission rows collapse the same way, the per-object GET/SET/DEL rows of the Messages path are untouched

#	phase	RedisCache op	keys	caller
1	pre	`async_get_cache`	`end_user_id:perf-enduser`	`async_get_cache <- async_get_cache`
2	pre	`async_get_cache`	`end_user_restricted_registry`	`async_get_cache <- async_get_cache`
3	pre	`async_get_cache`	`end_user_restricted_registry`	`async_get_cache <- async_get_cache`
4	pre	`async_set_cache`	`end_user_restricted_registry`	`_cache_registry_answer <- _fetch_and_cache_registry`
5	pre	`async_set_cache`	`end_user_id:perf-enduser`	`async_set_cache <- async_set_cache`
6	pre	`async_get_cache`	`<key-hash>`	`async_get_cache <- async_get_cache`
7	pre	`async_get_cache`	`<key-hash>`	`async_get_cache <- async_get_cache`
8	pre	`async_set_cache`	`<key-hash>`	`async_set_cache <- async_set_cache`
9	pre	`async_get_cache`	`team_id:<team>`	`get_team_object <- _get_hierarchical_router_settings`
10	pre	`async_delete_cache`	`team_id:<team>`	`get_team_object <- _get_hierarchical_router_settings`
11	pre	`async_set_cache`	`team_id:<team>`	`get_team_object <- _get_hierarchical_router_settings`
12	pre	`delete_cache`	`SYNC team_alias:perf-team`	`get_team_object <- _get_hierarchical_router_settings`
13	pre	`async_delete_cache`	`team_alias:perf-team`	`get_team_object <- _get_hierarchical_router_settings`
14	pre	`PIPELINE MGET+MGET`	`['<team>_<user>', '837848e9-670e-4b34-936e-156409a`	`_read_redis_rows <- _fill_from_redis`
15	pre	`async_get_cache`	`model_access_group_registry`	`cached_registry <- _load_bounded_registry`
16	pre	`async_get_cache`	`model_access_group_registry`	`cached_registry <- _load_bounded_registry`
17	pre	`async_set_cache`	`model_access_group_registry`	`_cache_registry <- _load_bounded_registry`
18	pre	`async_increment_pipeline`	`[{'key': 'spend:key:<key-hash>', 'increment_value': <cost>`	`_reserve_counters <- _reserve_reservable_counters`
19	pre	`PIPELINE SET+SET+SET+MGET+EVALSHA`	`['<user>', '<team>_837848e9-670e-4b34-936e-156409a`	`h_rate_limiter_script <- should_rate_limit`
20	pre	`PIPELINE EVALSHA+EVALSHA`	`['{api_key:<key-hash>}:window', '{api_key:ee876d0d5cd318`	`k_and_increment_by_n <- reserve_tpm_tokens`
21	pre	`async_get_cache`	`<key-hash>`	`ropic_messages_attempt <- make_call`
22	post	`async_set_cache`	`<key-hash>`	`async_add_cache <- _complete_cache_write_despite_cancellation`
23	post	`PIPELINE MGET`	`['spend:end_user:perf-enduser', 'spend:key:<key-hash>',`	`_collect_inflight <- _load`
24	post	`async_increment_pipeline`	`[{'key': 'spend:key:<key-hash>', 'increment_value': <cost>`	`ed <- increment_spend_counters`
25	post	`async_increment`	`gpt-dep-a:None:tpm:<min>`	`async_increment_cache <- async_log_success_event`
26	post	`async_batch_get_cache`	`['default_user_id:spend', 'tag:User-Agent: curl', 'tag:User-Agent: curl/7.81.0']`	`async_batch_get_cache <- _read_update_cache_values`
27	post	`EVALSHA rate-limiter check`	`['{api_key:<key-hash>}:tokens', '{user:837848e9-670e-4b3`	`_execute_token_increment_script <- async_increment_tokens_with_ttl_preservation`

The five pre-call round trips left on the chat path are separated by data dependencies, not by ownership: the reservation increments (row 2) need the spend read; the rate limiter (row 3) runs after auth and its result gates the TPM scripts (row 4); the response-cache key (row 5) depends on the deployment routing picked. Getting from 5 to 1 needs the speculative admission mode and the Lua check-and-reserve of LIT-8884, which this PR does not ship

Not measured here: a fleet load regime. These are single-request command counts on one box; the prod traces on LIT-8881 are what motivated the change

The fail-closed follow-up at b763921 makes a pipelined rate-limit or TPM group that Redis could not verify refund every applied group and then raise RateLimitUnverifiableError when fail-closed is on, instead of quietly falling back to in-memory counters. Dropping the _reject_if_rate_limit_unverifiable call turns both cases of test_fail_closed_rejects_when_a_pipelined_lua_group_cannot_be_verified red, and restoring it turns them green. 6c6354a only narrows the fake Redis types and pins the cooldown test timestamps

Re-run at the current head 6c6354a against real Redis, Postgres and Anthropic claude-sonnet-4-6, all requests 200:

$ bash run_arm2.sh head43407final ~/repos/litellm
warm 15 reads, chat_master 9, chat_vk 14, messages_vk 16   (redis total_reads_processed deltas)

Admin UI at `6c6354a`

The same virtual key request from the fixture, POST http://localhost:4000/v1/chat/completions with sk-perf-vk-0000000001 against Anthropic claude-sonnet-4-6, shows up on http://localhost:4000/ui/?page=logs as a Success row with its real spend, so the batched Redis path still records spend end to end

Live re-check at `ec4dd90`

This head keys the request batch by the cache's namespace as well as its connection settings, so two caches on one server with different prefixes each get their own pipeline and key prefix (test_caches_on_one_server_with_different_namespaces_keep_their_own_key_prefix, red when the namespace is dropped from _backend_key). A read-count harness, run from this checkout with PYTHONPATH=<checkout>, one warm request then one request per case, redis_reads_processed from INFO stats minus the three marker commands

$ bash run_arm2.sh <arm> <checkout>  # redis-cli flushall, proxy on :4000 with config_v2.yaml, real Redis 7, real Postgres, Anthropic claude-sonnet-4-6
key/generate 200
warm 200
warm redis_reads_processed=15
chat_master 200
chat_master redis_reads_processed=8
chat_vk 200
chat_vk redis_reads_processed=14
messages_vk 200
messages_vk redis_reads_processed=16

Every response body was pong from claude-group with usage populated, and the proxy log has no ERROR or Traceback lines

Type

🚄 Infrastructure

Caveats (if any)

Medium

Awaiting one result flushes every op declared so far, so a consumer that awaits early splits the pipeline. That is why reservation and the TPM scripts are separate round trips today
Auth write-backs now leave with the next flush (the rate limiter's pipeline in this fixture) or at request exit when nothing else flushes. The in-memory copy is updated at once as before; another worker reading Redis inside that window fills from the DB, as it would on a cache miss
On a RedisClusterCache every declared op still runs on its own, as before this PR. The cluster pipeline is per node and the existing per-op paths already group by slot

Low

A single rate-limit Lua call also rides the request batch now, so a request with one hash-tag group shares the round trip with the routing read. Outside a request scope (background jobs, SDK use) every caller runs exactly as before
Non-usage routing strategies only go through RoutingReadBatch when the request armed a cooldown prefetch; otherwise the router's plain cooldown read stays in charge, so callers and tests that replace litellm.router._async_get_cooldown_deployments still take effect
The proxy's cache and the router's cache share one pipeline when their connection settings compare equal as strings (the router receives its port as a string). Two caches with different settings get separate pipelines, flushed concurrently
redis.exceptions is imported lazily inside redis_batch.py, like redis_cache.py, so import litellm without redis installed keeps working

Final Attestation

The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/a666ccc35fde4ab799d68f3cc5bad523
Open in Devin Desktop: https://app.devin.ai/desktop/session/a666ccc35fde4ab799d68f3cc5bad523?variant=devin
Requested by: @yassin-berriai

devin-ai-integration · 2026-09-27T03:20:35Z

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

Disable automatic comment, CI, and merge conflict monitoring

CLAassistant · 2026-09-27T03:20:43Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

codecov · 2026-09-27T03:23:57Z

Codecov Report

❌ Patch coverage is 90.94828% with 42 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...itellm/proxy/spend_tracking/spend_counter_batch.py	29.62%	19 Missing ⚠️
litellm/caching/redis_batch.py	95.70%	11 Missing ⚠️
litellm/proxy/hooks/parallel_request_limiter_v3.py	90.00%	7 Missing ⚠️
litellm/proxy/auth/auth_object_prefetch.py	70.58%	5 Missing ⚠️

📢 Thoughts on this report? Let us know!

greptile-apps · 2026-09-27T03:26:33Z

Confidence Score: 5/5

[Critical risk] Batches Redis reads across auth, rate limiting, and routing in one pipeline per request.

The PR appears safe to merge based on the reviewed changes and resolved previous findings.

Summary

The PR adds request-scoped Redis pipelines for authentication, spend, rate-limit, and routing reads.

The changes since the previous review separate caches with different namespaces into distinct batches and add a regression test.
No new actionable issue was identified.

_{Reviews (15) · Last reviewed commit: "fix(caching): key the request Redis batc..."}

devin-ai-integration · 2026-09-27T03:58:06Z

@greptileai

devin-ai-integration · 2026-09-27T04:11:52Z

@greptileai

devin-ai-integration · 2026-09-27T04:20:55Z

@greptileai

devin-ai-integration · 2026-09-29T09:14:05Z

@greptileai

devin-ai-integration · 2026-09-29T09:16:17Z

@greptileai

devin-ai-integration · 2026-09-29T09:30:34Z

@greptileai

devin-ai-integration · 2026-09-29T09:41:12Z

@greptileai

yassin-berriai · 2026-09-29T10:28:44Z

bugbot run

cursor

Stale Bugbot comment from a previous run.

codspeed · 2026-09-29T22:19:29Z

Merging this PR will not alter performance

✅ 31 untouched benchmarks

_{Comparing litellm_redis_p2_request_batch (2de50b6) with main (e7460f1)}

…limit and routing reads RedisBatch: one pipeline per Redis backend for independently declared operations (MGET, GET, Lua scripts, INCRBYFLOAT, SET, DEL), a future per operation so each owner keeps its own fallback, Redis Cluster hash-slot fallback. A request-scoped batch middleware shares that pipeline across the auth identity reads and write-back, the spend counter MGET, the rate limiter Lua groups and the routing read. A rate-limit denial stands when another pipelined group fails; every pipelined group is refunded on rejection; local cooldowns win over the prefetch. The routing prefetch failure log line strips request line breaks (CodeQL py/log-injection) Resolves LIT-8882 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* chore(codeowners): drop UI, migration, and CODEOWNERS self owners (#43653) * chore(codeowners): drop UI and migration code owners Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(codeowners): drop CODEOWNERS self-owner line Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yuneng <yuneng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(fireworks_ai): route and list the auto, auto-instant and firerouter routers (#43641) * feat(fireworks_ai): route and list the auto, auto-instant and firerouter routers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(fireworks_ai): drive the router request test through an httpx MockTransport Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(fireworks_ai): let custom firerouter/<models> IDs inherit the firerouter row's capabilities Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(fireworks_ai): integration coverage for router short names forwarding tool_choice and reasoning_effort Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(fireworks_ai): assert tool definitions reach the router upstream Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat: add model leaderboard page (#43649) * feat(proxy): add model leaderboard analytics * feat: add model insights task and range constants * feat: record task type from task tags in model usage rollup * feat: serve 365 days of model insights by UTC date * test: cover task tag resolution in model usage rollup * test: update model insights range limit test to 365 days * chore: regenerate dashboard api types for model insights * feat: add model insights aggregation helpers * test: cover model insights aggregation helpers * feat: redesign model leaderboard with stacked bars, treemap and ranking * test: update model leaderboard view test * feat: mark model leaderboard as beta in sidebar * chore: sync schema.prisma copies from root * fix: only treat task: prefixed tags as model insight tasks * feat: add metric type for model insights ranking * fix: rank model insights by selected metric and scope detail queries to ranked deployments * test: plain tags are not model insight tasks * test: cover metric ranking, deployment scoping and rollup round trip * fix: build model insights weeks and halves from the requested date range * test: cover empty weeks and range-based change comparison * fix: refetch by metric, show load errors and ignore stale responses * test: cover metric refetch and error state * feat: define model insight tasks in a JSON file * feat: return task labels and categories from model insights * feat: load model insight tasks from JSON * refactor: validate rollup task tags against the JSON task list * feat: serve the task list with model insights * refactor: drop hardcoded task list from constants * build: ship model insight tasks JSON in the wheel * test: cover model insight task JSON * refactor: take task labels and categories from the API * test: pass task info to task tile builder * refactor: color treemap by API-provided category * test: include tasks in model leaderboard fixture * fix: make daily model usage migration idempotent * feat: bound the model insights task query size * fix: compute task breakdown independent of the chart metric * test: task breakdown is stable across chart metrics * chore: regenerate lazy openapi snapshot for model insights * chore: regenerate dashboard api types for model insights * fix: keep previous ranking dimmed while a new metric loads * test: cover stale metric state in model leaderboard * refactor: drop task row cap constant * fix: return the full task breakdown instead of a truncated one * test: task query is not truncated * feat: add task summary types for model insights * feat: summarise tasks server-side on a separate model insights endpoint * test: cover the model insights tasks endpoint * chore: regenerate lazy openapi snapshot for model insights tasks * chore: regenerate dashboard api types for model insights tasks * refactor: drop client-side task aggregation * test: remove client-side task aggregation tests * feat: load task breakdown separately from the chart metric * test: task breakdown is not refetched on chart metric change --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * fix(google_genai): preserve proxy_server_request in completion adapter (#43536) * fix(otel): send cache and reasoning tokens in langfuse usage_details (#43553) * fix(otel): send cache and reasoning tokens in langfuse usage_details The OTel V2 Langfuse mapper only sent input, output and total, so cache reads, cache writes and reasoning tokens never reached Langfuse. Emit them as input_cached_tokens, input_cache_creation and output_reasoning_tokens, and send input/output net of those buckets so Langfuse does not price the same tokens twice. Fixes #43542 * fix(otel): drop redundant comments from the usage_details change * fix(vertex_ai): forward the per-turn-control beta for per-message output_config (#43558) Claude Code attaches output_config to mid-conversation system messages and sends the per-turn-control-2026-07-01 beta with it. The Vertex beta map dropped that beta, so Vertex rejected the body with 'messages.N.output_config: Extra inputs are not permitted'. Forward the beta for vertex_ai, the way azure_ai already does, and add it on the Vertex Messages path whenever a message carries output_config. * fix(cost-map): add tool calling and reasoning flags, correct max output for nebius DeepSeek-V4.1-Flash (#43588) * fix(proxy): resolve model_group_alias in the zero-cost budget predicate (#43512) `_is_model_cost_zero()` reads a group's cost through `Router.get_model_group_info()`, which resolves `model_group_alias`, and then gates that on `_is_cost_explicitly_configured()`, which scanned `Router.model_list` for an exact `model_name` match. Alias names live only in `Router.model_group_alias` and are never `model_name` entries, so the scan found nothing and returned False. That False means "the zero cost was defaulted, not configured" (the sparse auto-registration gate added for #24770), so a model priced explicitly at 0 had budget enforced against it when requested through an alias, while the same deployment under its own name was exempt. Both names route to the same deployment and add nothing to spend. The two lookups in one function disagreeing is the bug, so they now share one resolution: `_is_cost_explicitly_configured()` resolves through `Router.get_model_list()`, the same alias-aware path `get_model_group_info()` takes. That also reaches a deployment which prices itself through its `model_info` block, whose cost-map entry lands under the deployment id. `_group_declares_explicit_cost()` was an alias-aware copy of this function, wired only into `model_has_no_cost_mapping()` and never into the budget path; its body is what `_is_cost_explicitly_configured()` now carries, and both callers share it so the two cannot drift apart again. `_has_ptu_flat_cost()` scanned `model_list` the same way and runs after the gate above, so resolving one without the other would let an aliased PTU group — explicit zero per-token price alongside a flat capacity cost — pass as free. It resolves the same way now. Tests cover the predicate and the request path it feeds: over-budget requests through `_should_skip_budget_checks()` into `common_checks()` for an aliased free model (allowed) and an aliased paid model (refused), the predicate for free, paid, PTU, hidden and dangling aliases, and `model_has_no_cost_mapping()` through an alias so the other caller of the shared check stays covered. Unchanged: priced groups (the predicate returns False before the gate), unmapped groups whose zero cost was defaulted (#24770), hidden aliases and aliases pointing at a nonexistent group (`get_model_group_info()` returns None for both, so the cost is unknown and budget is enforced), and non-aliased PTU groups. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor: clean up fresh tech debt from 2026-09-28 (#43674) * refactor: clean up fresh tech debt from 2026-09-28 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor: group leaderboard rows in one pass and wrap docstring at 120 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(types): replace Any with proven types in 7 files (#43704) * refactor(types): replace Any with proven types in 11 files Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(types): revert Any changes that broke existing callers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(types): drop prompt factory helper wrappers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover typing sweep surfaces Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): tighten sweep audit tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs(security): point readers to the security announcements mailing list signup (#43713) * docs(security): point readers to the security announcements mailing list signup Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs(security): formalize the security announcements wording Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs(security): tighten the best-effort sentence Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: oliver <oliver@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(rust): add shared llms wire type derives (#43730) * fix(router): stream /v1/messages lifecycle frames live when no fallback can take over (#43600) * fix(router): stream anthropic messages lifecycle frames live when no fallback can take over The /v1/messages streaming wrapper buffered message_start and content_block_start until the first content_block_delta and dropped pings behind buffered frames unconditionally, even for requests no fallback could ever recover. With adaptive thinking on Bedrock or Vertex the client saw no bytes for the whole thinking pass and hit read timeouts. Buffering now applies only while a fallback can still take over (generic or refusal chain resolving), and a ping is always forwarded live since it carries no lifecycle and keeps the connection alive. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): mirror every dispatcher fallback path in the anthropic stream gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): skip already-tried order levels in the anthropic stream gate Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): keep a transport-split ping behind buffered lifecycle frames instead of forwarding its head live Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(router): credit the #39566 branch this fix supersedes Co-authored-by: Radu Swigler <radu.porumba@gmail.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: mateo <mateo@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Radu Swigler <radu.porumba@gmail.com> * feat(cost-map): add baseten DeepSeek-V4.1-Flash-Fast (#43735) Price-Sync: litellm-providers Co-authored-by: berriai-litellm-provider-info-sync[bot] <328147090+berriai-litellm-provider-info-sync[bot]@users.noreply.github.com> * feat(guardrails): send a configured gateway_name from noma_v2 to Noma (#43678) * feat(guardrails): send a configured gateway_name from noma_v2 to Noma The noma_v2 guardrail accepts a gateway_name param, falling back to the NOMA_GATEWAY_NAME env var. The value is stripped, and when it is non-empty it goes out as a top-level gateway_name field on /litellm/guardrail. The param works for both guardrail: noma_v2 and guardrail: noma with use_v2, and it is appended after the existing constructor params so positional callers keep their meaning * chore(ui): regenerate OpenAPI snapshot and dashboard types for gateway_name The new noma_v2 gateway_name param shows up in the proxy OpenAPI spec, so the lazy snapshot and the generated dashboard types need regenerating * Update litellm/proxy/guardrails/guardrail_hooks/noma/noma_v2.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * chore(cost-map): add openai gpt-6.1-sol from the pricing page (#43738) Price-Sync: litellm-providers Co-authored-by: berriai-litellm-provider-info-sync[bot] <328147090+berriai-litellm-provider-info-sync[bot]@users.noreply.github.com> Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): callback credential canary slots C1-C3 and D5 (#43630) * test(integration): credential canary suite harness Adds tests/integration/security with canary generation and search, sweeps over the database, GET routes, client responses, sink doubles and Redis, an owned proxy rig, a sweep sensitivity self-test and the config deployment api_key slot. Registers the security group in run.py, the manifest and the CircleCI integration matrix. * test(integration): widen canary route sweep and harden the rig Enumerate lazily registered feature routers, call parameterized routes with placeholder ids, fail on routes that return no response, skip provider pass-through routes, add an explicit admin-only route allowance, let the sink double use a configurable token, inflate gzip members anywhere in a blob, sweep Redis before the route walk, and trap outbound connections from the owned proxy. * test(integration): descend into any decoded value that can still hold an encoded canary * test(integration): bound canary decoding by depth and decoded bytes * test(integration): scope log-table and spend-log reads to the scenario window * test(integration): sweep spend-log rows in the scenario date window * test(integration): keep spend-log date window summarized * test(integration): resolve deployment ids, scope paginated log lists, key allowances by slot * test(integration): expect 404 from the caller-scoped team membership route * test(integration): use the rig's own master key and expect 404 from submission lookups * test(integration): check the overridden rig key without assuming the default key is unknown * test(integration): callback credential canary slots C1-C3 and D5 Team callback, team callback_settings, config default_team_settings and key metadata.logging Langfuse secrets, a team Datadog dd_api_key, and request-body Langfuse keys (allow_client_side_credentials) must reach only their sink. Each scenario checks its sink received the canary as auth and that the marker is visible at the stored body, the Logs drawer route and the sink. Adds a unit test that the stored request body snapshot carries no callback parameter. * test(integration): give the callback sink waits a wider bound * test(integration): sweep provider requests for callback credentials * test(integration): request-path credential canary slots D1-D4 (#43307) * test(integration): credential canary suite harness Adds tests/integration/security with canary generation and search, sweeps over the database, GET routes, client responses, sink doubles and Redis, an owned proxy rig, a sweep sensitivity self-test and the config deployment api_key slot. Registers the security group in run.py, the manifest and the CircleCI integration matrix. * test(integration): widen canary route sweep and harden the rig Enumerate lazily registered feature routers, call parameterized routes with placeholder ids, fail on routes that return no response, skip provider pass-through routes, add an explicit admin-only route allowance, let the sink double use a configurable token, inflate gzip members anywhere in a blob, sweep Redis before the route walk, and trap outbound connections from the owned proxy. * test(integration): descend into any decoded value that can still hold an encoded canary * test(integration): bound canary decoding by depth and decoded bytes * test(integration): scope log-table and spend-log reads to the scenario window * test(integration): sweep spend-log rows in the scenario date window * test(integration): keep spend-log date window summarized * test(integration): request-path credential canary slots D1-D4 * test(integration): read the Logs drawer and spend-log filter for failed request rows * test(integration): check the marker in each failed row's spend-log filter; run header slots on chat-family routes * test(integration): run D2 on embeddings again; only the client-header slot runs on chat-family routes * test(integration): resolve deployment ids, scope paginated log lists, key allowances by slot * test(integration): pass the slot deployment's model_info id to the route sweep * test(integration): expect 404 from the caller-scoped team membership route * test(integration): use the rig's own master key and expect 404 from submission lookups * test(integration): check the overridden rig key without assuming the default key is unknown * test(integration): sweep proxy logs, metrics, a Datadog intake and the Logs drawer for credential canaries (#43306) * test(integration): credential canary suite harness Adds tests/integration/security with canary generation and search, sweeps over the database, GET routes, client responses, sink doubles and Redis, an owned proxy rig, a sweep sensitivity self-test and the config deployment api_key slot. Registers the security group in run.py, the manifest and the CircleCI integration matrix. * test(integration): widen canary route sweep and harden the rig Enumerate lazily registered feature routers, call parameterized routes with placeholder ids, fail on routes that return no response, skip provider pass-through routes, add an explicit admin-only route allowance, let the sink double use a configurable token, inflate gzip members anywhere in a blob, sweep Redis before the route walk, and trap outbound connections from the owned proxy. * test(integration): descend into any decoded value that can still hold an encoded canary * test(integration): bound canary decoding by depth and decoded bytes * test(integration): scope log-table and spend-log reads to the scenario window * test(integration): sweep spend-log rows in the scenario date window * test(integration): keep spend-log date window summarized * test(integration): sweep proxy logs, metrics, a gzip Datadog intake and the Logs drawer for credential canaries * test(e2e): treat an unset prompt-storage setting as unset and restore it * test(integration): name the Datadog sink slot G1d * test(e2e): search the Logs page for base64 forms of the deployment key * test(integration): resolve deployment ids, scope paginated log lists, key allowances by slot * test(integration): pass the resolved deployment id to the Datadog route sweep * test(integration): expect 404 from the caller-scoped team membership route * test(integration): use the rig's own master key and expect 404 from submission lookups * test(integration): check the overridden rig key without assuming the default key is unknown * chore(cost-map): add azure and openrouter gpt-6.1-sol rows (#43744) Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): classify every credential-bearing param for the canary suite (#43298) * test(proxy): classify every credential-bearing param for the canary suite * test(proxy): classify gcs_path_service_account as secret, run registry in auth-checks shard, check slot ids at import * test(proxy): use one generic slot id for callback and request-body credential params * test(proxy): name a canary slot only for params an integration test plants * test(proxy): classify the SigNoz callback params * test(proxy): move the slot sync note into the module docstring * test(proxy): model unplanted credential params as their own classification * test(security): classify request-body api_key as unplanted until D1 exists; check registry slots against the harness * test(security): classify request-body api_key under slot D1 * test(security): classify Langfuse and Datadog callback secrets under slots C1 and C3 * fix(cost-map): lower fireworks up-to-4b size tier to the pricing page price (#43740) Price-Sync: litellm-providers Co-authored-by: berriai-litellm-provider-info-sync[bot] <328147090+berriai-litellm-provider-info-sync[bot]@users.noreply.github.com> * fix(cost_calculator): bill chat per-second pricing once with a new cost_per_second field (#43614) * feat(cost_calculator): add cost_per_second for chat per-second pricing Keep legacy input_cost_per_second and output_cost_per_second as aliases for chat, completion, embedding and responses. When both legacy fields are set, input_cost_per_second wins Move Bedrock commitment rows to cost_per_second so they bill once Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(cost_calculator): drop legacy per-second fields from chat paths Keep Azure chat token pricing generic and update inert Voxtral rates and SageMaker examples Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost_calculator): recognize output-only per-second rates Include output_cost_per_second when checking whether a deployment cost entry has pricing so output-only legacy aliases remain attached to the deployment during cost selection Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(pricing): cover cost_per_second and legacy per-second aliases through the proxy Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(cost_calculator): drop output_cost_per_second as a chat per-second alias Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(cost_calculator): restore output_cost_per_second as a chat per-second fallback Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost-map): keep input_cost_per_second on bedrock commitment rows for older clients Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(cost-map): add openai gpt-6-astra ultrafast tier prices from the pricing page (#43745) * chore(cost-map): add openai gpt-6-astra ultrafast tier prices from the pricing page Price-Sync: litellm-providers * feat(cost): support openai ultrafast tier fields in the model catalog --------- Co-authored-by: berriai-litellm-provider-info-sync[bot] <328147090+berriai-litellm-provider-info-sync[bot]@users.noreply.github.com> Co-authored-by: Kerry <kerry@berri.ai> * fix(autorouter): compare historical and new savings consistently (#43348) * fix(autorouter): compare historical and new savings consistently * fix(autorouter): reject comparisons if request counts changed * fix(router): restore eligible LLM and classification breakdown * fix(router): avoid ambiguous baseline labels for partial comparisons * fix(model-prices): align Azure, Bedrock, Copilot, Gemini, Groq, OpenAI and OpenRouter entries with official docs (#43598) * fix(model-prices): correct azure/eu/gpt-6-astra to Data Zone rates Co-authored-by: rain <1504569896@qq.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(model-prices): align groq, gemini and openai entries with official docs Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(model-prices): roll in verified Vertex, Gemini, OpenRouter and Azure AI registry fixes Absorbs the fields from #43609, #43666, #43671 and #43644 that match the provider's own docs or price API today, and adds a cost test for the azure/eu/gpt-6-astra Data Zone tiers Co-authored-by: bunnysayzz <stfuazzo@gmail.com> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(model-prices): add Copilot, Bedrock Kimi K3, Gemini Robotics and OpenRouter values from official sources Co-authored-by: Michal Formanek <michal.formanek@generaliceska.cz> Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: rain <1504569896@qq.com> Co-authored-by: bunnysayzz <stfuazzo@gmail.com> Co-authored-by: Michal Formanek <michal.formanek@generaliceska.cz> * fix(guardrails): enable explicit PANW MCP output scanning (#43109) * fix(guardrails): declare post_mcp_call for PANW Prisma AIRS Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(guardrails): exercise post_mcp_call_hook dispatch in PANW tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(model-catalog): add fal_ai resolution-tiered image cost fields Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(guardrails): keep post_mcp_call opt-in for PANW Prisma AIRS Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(model-catalog): add fal_ai resolution-tiered image cost keys to cost map schema Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: joshua <joshua@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> * refactor(rust): orchestrate Messages route execution (#43719) Co-authored-by: Yujong Lee <yujong@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(streaming): keep the served service_tier on streamed chunks and spend rows (#42870) * fix(streaming): keep the provider's served service_tier on streamed chunks and spend rows Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(streaming): satisfy type-discipline and strict ruff budgets Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(streaming): stamp the served service_tier on every Responses bridge chunk Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic-adapter): expose streamed chunks so disconnects bill partial spend Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(service-tier): cover anthropic and responses served-tier billing paths Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic-adapter): return a chunks-exposing stream so disconnects bill partial spend Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(service-tier): bill disconnects through the router's anthropic stream wrapper Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style: apply ruff format to the anthropic stream changes Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(coverage): ignore delegating properties the ast scan cannot see Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style: keep the cast-ok reasons on the cast call line Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(integration): cover served service_tier billing for streamed chat and messages, complete and disconnected Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(anthropic-cache): delegate chunks/messages/model through the messages stream cache writer Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(streaming): keep service_tier on OpenAI-compatible parsed chunks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(streaming): parameterize delegated chunks and messages types Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(tests): follow the anthropic pass_through rename after merging main Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(anthropic): drain the logging worker between response cache tests Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(spend): cover azure, databricks, responses bridge and gemini served tiers in the stream billing integration test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(databricks): keep the served service_tier on streamed chunks and bill it Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(databricks): type the served service_tier chunk without a loose kwargs dict Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost): bill the served service_tier over the requested one Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(cost): drop explanatory comment from the tier resolution Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: kerry <kerry@berri.ai> * feat(bedrock): add openai gpt-6.1-sol global and base rows (#43758) Price-Sync: litellm-providers Co-authored-by: berriai-litellm-provider-info-sync[bot] <328147090+berriai-litellm-provider-info-sync[bot]@users.noreply.github.com> * perf(router): fetch cooldown state and usage counters in one Redis round trip (#43320) * perf(router): fetch cooldown state and usage counters in one Redis round trip The cooldown filter (CooldownCache) and usage-based-routing-v2 selection (LowestTPMLoggingHandler_v2) each issued their own MGET on every request because they live in different objects. RoutingReadBatch fetches both key sets through DualCache.async_batch_get_cache_shared while the healthy deployments are resolved and hands the usage slice to the strategy, so selection does not read again. Each cache keeps its own memory tier, throttling, reservation rollback and circuit-breaker handling, and the strategy falls back to its own read when the prefetch does not cover its keys. simple-shuffle keeps reading only cooldowns. aresponses no longer issues a second, blocking response-cache read from the worker thread that runs the sync wrapper. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(caching): keep per-cache tier failures inside the shared batch read Wrap the memory-tier prepare and backfill steps of DualCache.async_batch_get_cache_shared so a failing tier degrades that cache's read to None the way async_batch_get_cache does, instead of escaping into routing. Drop the aresponses sync-cache guard: for native Responses models the worker-thread read is the one whose key matches the write, so skipping it broke cached /v1/responses replays. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(router): rename usage key builder so the async cache-call check reads it as a key helper Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(alerting): narrow daily-report cache values before numeric comparison Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(caching): type the shared batch-read helpers and merge Redis results without mutation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style: fix import sort in test_dual_cache Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * refactor(caching): flatten shared batch read keys without a stacked comprehension Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * perf(proxy): hold one spend counter batch across admission and across post-call accounting (#43369) Auth's spend counter MGET scope spans common checks, model budget check and reservation; reservation increments go out as one pipeline; post-call reconcile adjustments ride the ordinary increment pipeline and update_cache uses one batched read. Over-budget reservation counters are charged one at a time so a rejection never touches the counters after it; post-call counter keys are derived from ids without validating a UserAPIKeyAuth. Resolves LIT-8881 Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * perf(responses): run aresponses through the async wrapper so the cache is read once (#43769) * perf(responses): run aresponses through the async wrapper so the cache is read once aresponses() sets kwargs["aresponses"] = True and runs the decorated sync responses() on an executor, but _is_async_request() did not recognise that flag, so the sync wrapper did a second cache lookup on the executor thread with a differently ordered cache-key input. Every /v1/responses request paid two cache GETs against two different keys. Recognising aresponses in _is_async_request() leaves the async wrapper as the only cache reader and writer for the async path, one GET per request, same key on read and write Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(caching): let a responses cache entry cover aresponses so responses-only configs keep caching Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): bind Claude Code background sessions to their auto-router (#43767) Claude Code background sessions (claude --bg) stamp x-app: cli-bg on every request, including main-loop turns. The session router binding only accepted x-app: cli, so a background session never bound and its subagents' concrete-model calls bypassed the router. The binding write already requires the requested model to resolve to a pre-routing strategy, so background side calls naming plain models still never bind. Since f6eff1bde0 removed the clear path, the x-app check guarded nothing else. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> * feat(bedrock): add openai.gpt-6.1-sol us geo cris and Mantle rows (#43763) Co-authored-by: kerry-berri <kerry@berri.ai> * fix(proxy): recover session key owners from daily spend for usage attribution (#43642) * fix(proxy): recover daily spend key owners Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): simplify daily spend owner recovery Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(proxy): format daily activity metadata Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover recovered owner metadata merge Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): bound the daily spend owner lookup with the statement timeout * test(integration): audit the daily activity key owner fallback on every usage route Thirty five integration cells under tests/integration/spend cover the daily spend owner fallback on all nine daily activity routes and /usage/ai/chat: the happy path per route, the unanimity rules (two users, blank and null rows, an owner the user table lacks, live and deleted keys with and without their own user, a spend log alias), a non admin reader, an invalid key, a 5 KB key, a locked LiteLLM_DailyUserSpend, 300 keys of one team, repeated reads, a second user landing between reads, a concurrent burst across the unified endpoints, a killed worker, and a proxy restart The traffic cells ignore the GET /v1/models call the proxy's five minute token limit refresh makes to every registered OpenAI compatible deployment, since it lands on a test's provider wire whenever the refresh instant falls inside the test --------- Co-authored-by: jesus <jesus@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> * chore(cost-map): take azure context limits from models-sold-directly (#43759) Price-Sync: litellm-providers Co-authored-by: berriai-litellm-provider-info-sync[bot] <328147090+berriai-litellm-provider-info-sync[bot]@users.noreply.github.com> * perf(proxy): one request-scoped Redis pipeline for auth, spend, rate-limit and routing reads (#43407) RedisBatch: one pipeline per Redis backend for independently declared operations (MGET, GET, Lua scripts, INCRBYFLOAT, SET, DEL), a future per operation so each owner keeps its own fallback, Redis Cluster hash-slot fallback. A request-scoped batch middleware shares that pipeline across the auth identity reads and write-back, the spend counter MGET, the rate limiter Lua groups and the routing read. A rate-limit denial stands when another pipelined group fails; every pipelined group is refunded on rejection; local cooldowns win over the prefetch. The routing prefetch failure log line strips request line breaks (CodeQL py/log-injection) Resolves LIT-8882 Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): scope OpenAPI listings to the exact server prefix and drop upstream OAuth metadata when a server is saved (#43608) * fix(mcp): key discovery caches per caller correctly and drop stale caches on server updates Discovery-list cache identity now uses the hashed token instead of the raw api_key and treats MCPJWTSigner-signed servers as per caller. Server definition changes also drop the cached upstream OAuth metadata. OpenAPI listings look tools up under the normalized registry prefix with the separator, so an overlapping sibling prefix no longer leaks into the list. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): keep the discovery cache digest call unchanged so CodeQL matches the existing alert Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): guard OAuth metadata cache writes with a per-server generation and drop unproven per-caller discovery keys An upstream metadata fetch that started before a server edit could store its stale reply after invalidate_oauth_metadata_cache ran. Invalidation now bumps a per-server generation and the fetch only stores when the generation it captured before I/O is unchanged. The MCPJWTSigner-based per-caller discovery classification and the api_key to token key change had no reproduction (the signer only injects on tools/list, and UserAPIKeyAuth hashes api_key in place), so both go back to the merge-base behavior. Integration coverage under tests/integration/mcp: overlapping OpenAPI aliases, a config-declared server name with a space, OAuth metadata refetch after a save, and the in-flight stale-write race Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): keep OAuth metadata generations only while a fetch is in flight Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): count queued OAuth metadata fetchers so invalidation survives lock handoff Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(mcp): keep a held OAuth metadata lock registered even when no fetcher slot claims it Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(mcp): prove a peer worker drops stale upstream OAuth metadata after a save elsewhere Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * perf(proxy): one post-call Redis pipeline per backend for spend, rate-limit, routing and response-cache writes (#43779) Post-call owners declare into one request-scoped RedisBatch per Redis backend: spend counter increments and reservation reconciliation, rate-limit token Lua updates and refunds, parallel-slot release (freed locally at once), deployment TPM, and compatible async response-cache SETs. The batch is sent once the success and failure callbacks have run, or on a deadline, and pending batches are drained at shutdown before Redis disconnects. nx writes, non-Redis caches and calls outside a request stay direct; numeric string TTLs keep the direct-path coercion. Resolves LIT-8883 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: yassin <yassin@berri.ai> * fix(bedrock): set gpt-6.1-sol max output tokens to 131072 (#43782) Price-Sync: litellm-providers Co-authored-by: berriai-litellm-provider-info-sync[bot] <328147090+berriai-litellm-provider-info-sync[bot]@users.noreply.github.com> * perf(proxy): refresh auth management objects through the request Redis pipeline (#43776) Identity objects (key, end user) load through the request MGET and their write-backs, the registry reads and the management-object SETs ride the request pipeline. A team refresh invalidates its alias with a pipelined DEL instead of a synchronous DEL plus a duplicate async one, and an MGET miss is remembered so no per-key GET follows it in the same request. Resolves LIT-9012 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: yassin <yassin@berri.ai> * feat(agents): add identity storage and validation contracts (#43720) * feat(agents): identity storage and contracts * fix(agents): cache positive identity lookups with fresh policy checks * test(agents): include identity attribution in spend fixture --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> * bump: litellm-enterprise 0.1.71 -> 0.1.72, litellm-proxy-extras 0.4.102 -> 0.4.103, litellm 1.104.0 -> 1.105.0 (#43789) * fix(auth): give UI/CLI session tokens their own AES-GCM context and header-safe shape (#43790) * refactor(auth): bind UI/CLI session tokens to their own AES-GCM context UI and CLI session tokens are now always encrypted with AES-256-GCM and a fixed session associated-data value, and the session-token check only accepts AES-GCM values carrying that same value. Stored secrets keep their current encryption and decrypt unchanged, so nothing needs migrating. encrypt_value_helper and decrypt_value_helper take an optional aad. XSalsa20 cannot bind associated data, so an AAD-bound value is always written as AES-256-GCM, and an AAD-bound decrypt refuses the legacy format. Session tokens issued before the upgrade stop validating, so UI and CLI users sign in once more after upgrading. * test(e2e): cover real SSO login through the dashboard and the lite CLI Adds two specs under tests/e2e/ui/oidc, run by playwright.oidc.config.ts against a live Keycloak stack. The dashboard spec checks that the SSO session authorizes the Virtual Keys and Models data requests. The CLI spec runs a real lite login in an isolated HOME with the keyring disabled, then lists models and sends one chat completion with the stored session. The main Playwright config now ignores oidc/. * fix(auth): encode UI/CLI session tokens as unpadded base64url Session tokens carried the v2:gcm: storage prefix and base64 padding. Basic-auth parsers split on the first colon and browsers reject ':' and '=' in WebSocket subprotocols, so Langfuse pass-through and the realtime playground could not use them Tokens are now plain unpadded base64url, the same header-safe shape as any bearer token * fix(auth): prefix UI/CLI session tokens with litellm_login_ A prefix-less token starts with sk- about once in 262,144 logins and is then routed as a virtual key, so that login gets a 401. The prefix also makes session tokens easy to spot in logs The prefix doubles as the token's AES-GCM associated data, so the visible kind and the encrypted kind cannot disagree --------- Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> * chore(deps): bump pyjwt, moment and brace-expansion to clear osv-scan (#43792) pyjwt 2.13.0 -> 2.14.0 (uv.lock only, pyproject floor unchanged), moment 2.30.1 -> 2.31.0 and the brace-expansion override 5.0.9 -> 5.0.12 in the dashboard. oauthlib's only fixed release (4.0.0, 2026-09-28) is still inside the 3-day uv exclude-newer cooldown, so its two findings are ignored until 2026-10-02 * fix(ui): surface x-litellm-call-id in Logs search, table and drawer (#42436) * fix(ui): surface x-litellm-call-id in Logs search, table and drawer Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(ui): regenerate api types for spend logs search description Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(e2e): drop redundant comments from the call id logs helpers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * style(e2e): format logs call id helper and spec Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): keep one id per Logs row, move x-litellm-call-id to hover and drawer The Request ID cell shows only request_id again. When the row's litellm_call_id differs, the cell tooltip lists it as x-litellm-call-id with its own copy button, and the drawer header labels the second line x-litellm-call-id: instead of the call id caption. Stacking two ids in every row made the column noisy for the common case where the viewer only needs the row they searched for. * test(e2e): cover the Request ID tooltip hover and copy path Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: poll the clipboard after the tooltip copy and drop a jsdom aside The e2e read navigator.clipboard right after the click, so a slow async write could fail the check even though copy works. The unit test's fireEvent choice (jsdom has no layout, so a real pointer move off the trigger closes the tooltip before the click lands) is documented here instead of inline. --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> * fix(proxy): look up hashed key names with two spend log rows per key (#43656) * fix(proxy): look up hashed key names with two spend log rows per key The spend-log fallback for keys missing from the key table read every row per key to check that all named rows agreed, which passed the 5s statement timeout on busy keys even with the (api_key, startTime) index. Probe only the oldest and newest named row per key, so the lookup stays two index reads per key however much the key logged. * fix(proxy): cap each spend log name probe at 100 rows per key * fix(proxy): bound the newest-row probe at where the oldest probe stopped The newest-row probe now starts at the row where the oldest-row probe gave up, so a key with under 200 rows in the window is read once instead of twice, and the lookup transaction turns bitmap scans off so the planner walks the (api_key, startTime) index instead of every row of a busy key when statistics or the visibility map are stale. * test(integration): add spend log alias probe cells for the daily activity routes --------- Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com> * fix(cost-map): add deprecation_date to two together_ai nvidia rows (#43809) Price-Sync: litellm-providers Co-authored-by: berriai-litellm-provider-info-sync[bot] <328147090+berriai-litellm-provider-info-sync[bot]@users.noreply.github.com> * test(ci): refresh retired OpenAI tool-call models (#43676) * fix(ui): keep MCP permissions visible after key, team and MCP server saves (#43810) * fix(ui): keep MCP permissions visible after key, team and MCP server saves Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(proxy): type the object_permission include as a prisma TypedDict Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): do not block key save confirmation on cache refetch Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: ryan <ryan@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(ci): repair MCP Responses and budget fixtures (#43788) * test(ci): repair MCP Responses and budget fixtures * test(auth): verify delegated budget changes persist * test(bedrock): accept regional aliases that inherit Converse routing (#43785) * test(bedrock): accept regional aliases that inherit Converse routing * test(bedrock): cover regional alias metadata independently of catalog * fix(params): filter internal traceback flag from provider requests (#43783) * perf(router): honour the cooldown read interval in the routing prefetch (#43815) Resolves LIT-9043 Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): carry per-request routing reads on context variables instead of public method kwargs (#43814) Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(bedrock): add beta header for output config in message (#43778) * refactor: clean up fresh tech debt from 2026-09-29 (#43830) * refactor: clean up fresh tech debt from 2026-09-29 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(routing): pin usage-based routing Redis reads through the proxy and SDK Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): settle the shared logging worker before recording shadow callbacks (#43847) * test(router): settle the shared logging worker before recording shadow callbacks Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(router): always stop the shared logging worker after settling it Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(model_prices): add Gemini Veo, Mistral and Azure Claude 4.5 deprecation dates (#43857) * fix(cost_calculator): bill ultrafast prompts above 272k at the ultrafast long-context rates (#43764) * refactor(rust): centralize bridge execution wrappers (#43871) Co-authored-by: Yujong Lee <yujong@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * chore(cost-map): add openai gpt-image-2.5 batch prices from the pricing page (#43869) Price-Sync: litellm-providers Co-authored-by: berriai-litellm-provider-info-sync[bot] <328147090+berriai-litellm-provider-info-sync[bot]@users.noreply.github.com> * chore(cost-map): add fireworks priority prices for ember-1, nemotron and glm 5.3 us rows (#43811) Price-Sync: litellm-providers Co-authored-by: berriai-litellm-provider-info-sync[bot] <328147090+berriai-litellm-provider-info-sync[bot]@users.noreply.github.com> * fix(proxy): attribute completed batch cost rows to /batches in daily activity (#43870) * fix(proxy): attribute completed batch cost rows to /batches in daily activity Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(spend): wait for priced batch tokens before asserting team endpoint activity Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: kerry <kerry@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(router): strip encrypted reasoning the pinned deployment cannot decrypt (#43781) Co-authored-by: yassin <yassin@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(cost_calculator): stop copying optional_params into response hidden params (#43637) * fix(cost_calculator): stop copying optional_params into response hidden params * test(cost_calculator): assert the stored spend-log request and logging payload carry no forwarded credentials * feat(pricing): add vertex_ai gemini-3.8 flash tts rows (#43876) Price-Sync: litellm-providers Co-authored-by: berriai-litellm-provider-info-sync[bot] <328147090+berriai-litellm-provider-info-sync[bot]@users.noreply.github.com> * fix(proxy): keep request-body credentials out of stored spend-log requests (#43635) * fix(proxy): keep request-body aws credentials out of stored spend-log requests * fix(proxy): redact every credential-named request-body field in stored spend-log requests Replace the hard-coded AWS key check in the spend-log request-body sanitizer with SensitiveDataMasker's key classification, so Azure, Vertex, watsonx, OCI, GigaChat, Gemini and header credentials are redacted too. Proxy-stamped key identity metadata is kept. * fix(proxy): keep request identifiers named like keys in stored spend-log requests * refactor(proxy): drop the AWS-only snapshot exclusion now that spend-log redaction is name-based * refactor(proxy): use SensitiveDataMasker's key classification without an exclusion list * refactor(proxy): always redact credential-named fields in stored spend-log payloads * fix(ui): right-align money and count columns across tables (#37889) * fix(ui): right-align money and count columns across tables Make numeric the one alignment token for the three shared table wrappers (DataTable, MemberTable, SimpleTable) via NUMERIC_CELL_CLASS, and flag every money, cost, and bare-count column that was still left-aligned. Raw ui/table usages that render spend or budgets get the same class on their header and cell. * test(ui): render the organizations alignment test with providers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(ui): query alignment cells by role instead of DOM traversal Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(agents): enforce authoritative agent permissions (#43721) * feat(agents): authoritative permissions * fix: enforce authoritative managed agent permissions * fix(agents): only consult the identity store for managed targets is_agent_allowed entered the identity-store path whenever a prisma client was configured, so an ordinary agent paired with an internal user returned 503 instead of 200. Classify the target from the registry first and fall back to the store only when the registry has no entry, so an unmanaged target never depends on the store being reachable. * fix(agents): gate the managed path on an admitted policy object Ten call sites branched on `managed_agent_policy is not None`, which any MagicMock attribute satisfies, so the managed path fired on unmanaged subjects and died in Pydantic validation as a 503. Route every check through a shared helper that requires a real AgentResponse. * test(mcp): stub the writer replica the fresh-policy reads use reload_admitted_user now passes check_db_only through to get_user_object, so the user row is read from writer_db. Point the mocks at the replica the code actually reads and give each parametrized case its own user id. * fix(agents): cap a managed agent at the invoking team's agents resolve_agent_access returned the managed policy's grants before the agent_caller ceiling was applied, so a managed agent acting on behalf of a user reached agents that user's team was never granted. Intersect with the caller ceiling the unmanaged path already honours. * fix(agents): restore token narrowing and scope the private-access suppressions The managed-model check lost its valid_token narrowing when it moved to the shared helper. Make the caller-access resolver public rather than reaching into it from module scope, and give each remaining private access a reason. * docs(agents): drop the comment claiming admins skip the A2A permission check The check has never had an admin bypass on this path, so the comment described behaviour the code does not implement. * test(proxy): stub the writer reads and restore the MCP manager singleton Fresh-policy user lookups read writer_db, so the team and rest-endpoint mocks stubbed a replica the code no longer reads, and the dashboard session fake still had the pre-kwarg signature. The manager reload also rebound global_mcp_server_manager in every MCP module without restoring it, leaking an empty manager into later files. * style: sort imports under the litellm package ruff config * fix(mcp): cap a managed agent's servers and tools at the invoking caller managed_agent_servers and managed_agent_tools returned the agent's own grants without the agent_caller ceiling the unmanaged resolvers apply, so a managed agent reached MCP servers and tools the echoed caller could not. Call the existing ceiling helpers on both axes. * refactor(mcp): return the caller-capped tools without an interim list The ceiling helper already returns a sequence, so materializing it into a list added a mutable collection for nothing. Sort at the return sites instead, which also makes the tool order stable across both branches. * fix(agents): preserve actor ceilings during managed target checks * fix(agents): keep managed permission ceilings authoritative * fix(mcp): fail closed on authoritative caller team outages --------- Co-authored-by: Joshua Valluru <326636767+joshua-berri@users.noreply.github.com> * fix(proxy): strip caller credentials from websocket passthrough (#43855) * fix(proxy): strip caller credentials from websocket passthrough Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(proxy): cover configured x-api-key in websocket passthrough credential test Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: oliver <oliver@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(ui): render access group MCP and agent selections as wrapping chips (#41228) * fix(ui): render access group MCP and agent selections as wrapping chips Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(ui): cover 20 selected MCP servers rendering as separate chips Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(ui): cover MCP and agent chip selection in access group create dialog Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: ryan-crabbe-berri <ryan@berri.ai> * fix(responses): scan and mask top-level instructions with guardrails (#43629) * fix(responses): scan and mask top-level instructions with guardrails The Responses guardrail translation handler put a non-empty top-level instructions field into structured_messages as a system row but never into the flat texts list, so guardrails that scan texts skipped it, flat-text masking could not rewrite it, and PANW latest-only selection failed its alignment guard whenever instructions were present. Seed texts with the instructions row, carry that offset into the flat-text write-back so a rewritten row lands on data["instructions"], and account for the leading row in the PANW Responses alignment. Resolves LIT-8931 Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): reject empty guardrail rewrites instead of forwarding raw input An explicit texts=[] answer from a guardrail now fails the count check and raises UnappliableRequestRewrite like any other misaligned rewrite; only a missing texts key means no rewrite. Types the out-param as dict[str, object] and adds integration coverage for instructions blocking, masking, empty instructions, tool loops, latest-only and concurrent workers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): type the texts-replacing guardrail helper explicitly Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): honor skip_system_message_in_guardrail for instructions and system input items Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): cover skip_system_message_in_guardrail on the live proxy Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): keep skipped rows through full-coverage rewrites and align latest-only with skip_system Trust a guardrail's structured_messages_cover_full_request claim only when it returns as many rows as the full normalized request, otherwise merge the scoped rows back so skipped instructions and system items survive the write-back. Make PANW's Responses reasoning alignment skip-aware so latest-only still picks the latest user turn when system content is excluded from texts. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): annotate new guardrail tests with return types Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(responses): treat an empty guardrail texts answer as no rewrite like chat completions Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test(responses): type the guardrail test doubles explicitly Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: yucheng <yucheng@berri.ai> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * feat(traces): add Rust storage foundation (#43819) * wip * feat(traces): establish shared Rust storage foundation * fix(traces): escape ClickHouse text parameters * test(traces): exercise response cap with bounded strings * fix(traces): remove unnecessary lint expectation * fix(traces): encode ClickHouse timestamp units in Rust * test(traces): mark exception match as a regex * refactor(traces): execute schema setup in Rust * refactor(traces): use shared logging execution wrapper * docs(traces): replace foundation README with boundary rules * fix(traces): use current bridge execution facade * fix(traces): account for protocol cast in lint budget Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Devin AI <158243242+…

devin-ai-integration Bot assigned yassin-berriai Sep 27, 2026

devin-ai-integration Bot had a problem deploying to e2e-changed September 27, 2026 03:20 Error

devin-ai-integration Bot requested a review from tin-berri September 27, 2026 03:20

devin-ai-integration Bot had a problem deploying to e2e-changed September 27, 2026 03:25 Error

greptile-apps Bot reviewed Sep 27, 2026

View reviewed changes

Comment thread litellm/proxy/hooks/parallel_request_limiter_v3.py

Comment thread litellm/caching/redis_batch.py Outdated

Comment thread litellm/router_utils/routing_read_batch.py Outdated

devin-ai-integration Bot had a problem deploying to e2e-changed September 27, 2026 03:29 Error

devin-ai-integration Bot had a problem deploying to e2e-changed September 27, 2026 03:57 Error

greptile-apps Bot reviewed Sep 27, 2026

View reviewed changes

Comment thread litellm/proxy/hooks/parallel_request_limiter_v3.py

Comment thread litellm/router.py Outdated

devin-ai-integration Bot had a problem deploying to e2e-changed September 27, 2026 04:11 Error

greptile-apps Bot reviewed Sep 27, 2026

View reviewed changes

Comment thread tests/unit/caching/test_request_redis_batch_pre_call.py Outdated

Comment thread tests/unit/caching/test_request_redis_batch_pre_call.py Outdated

Comment thread tests/unit/caching/test_request_redis_batch_pre_call.py Outdated

devin-ai-integration Bot had a problem deploying to e2e-changed September 27, 2026 04:20 Error

devin-ai-integration Bot mentioned this pull request Sep 27, 2026

perf(proxy): one post-call Redis pipeline per backend for spend, rate-limit, routing and response-cache writes #43424

Merged

6 tasks done

devin-ai-integration Bot had a problem deploying to e2e-changed September 29, 2026 09:14 Error

devin-ai-integration Bot had a problem deploying to e2e-changed September 29, 2026 09:16 Error

devin-ai-integration Bot had a problem deploying to e2e-changed September 29, 2026 09:18 Error

devin-ai-integration Bot had a problem deploying to e2e-changed September 29, 2026 09:26 Error

devin-ai-integration Bot had a problem deploying to e2e-changed September 29, 2026 09:41 Error

cursor Bot reviewed Sep 29, 2026

View reviewed changes

Comment thread litellm/proxy/hooks/parallel_request_limiter_v3.py

devin-ai-integration Bot mentioned this pull request Sep 29, 2026

perf(proxy): request-scoped RedisBatch plan for pre-call identity, spend, rate limit and routing reads #43403

Closed

6 tasks done

devin-ai-integration Bot force-pushed the litellm_redis_p1_spend_batch_request_scope branch from 178bd46 to a51b69e Compare September 29, 2026 21:33

devin-ai-integration Bot force-pushed the litellm_redis_p2_request_batch branch from ec4dd90 to 13e171c Compare September 29, 2026 21:33

devin-ai-integration Bot deployed to e2e-changed September 29, 2026 21:33 Active

Base automatically changed from litellm_redis_p1_spend_batch_request_scope to main September 29, 2026 22:05

yassin-berriai requested a review from a team September 29, 2026 22:05

devin-ai-integration Bot force-pushed the litellm_redis_p2_request_batch branch from 13e171c to fada595 Compare September 29, 2026 22:07

devin-ai-integration Bot deployed to e2e-changed September 29, 2026 22:07 Active

github-advanced-security Bot found potential problems Sep 29, 2026

View reviewed changes

Comment thread litellm/router.py Fixed

devin-ai-integration Bot had a problem deploying to e2e-changed September 29, 2026 23:02 Error

devin-ai-integration Bot mentioned this pull request Sep 29, 2026

perf(proxy): refresh auth management objects through the request Redis pipeline #43776

Merged

4 of 6 tasks

devin-ai-integration Bot force-pushed the litellm_redis_p2_request_batch branch from 1476a1c to da1e602 Compare September 29, 2026 23:13

devin-ai-integration Bot had a problem deploying to e2e-changed September 29, 2026 23:13 Error

yassin-berriai approved these changes Sep 29, 2026

View reviewed changes

devin-ai-integration Bot force-pushed the litellm_redis_p2_request_batch branch from da1e602 to 2de50b6 Compare September 29, 2026 23:30

yassin-berriai enabled auto-merge (squash) September 29, 2026 23:30

devin-ai-integration Bot requested a deployment to e2e-changed September 29, 2026 23:30 Waiting

yassin-berriai disabled auto-merge September 29, 2026 23:30

yassin-berriai merged commit ffb15f9 into main Sep 29, 2026
94 of 97 checks passed

devin-ai-integration Bot mentioned this pull request Sep 29, 2026

perf(proxy): one post-call Redis pipeline per backend for spend, rate-limit, routing and response-cache writes #43779

Merged

6 tasks done

mateo-berri mentioned this pull request Sep 29, 2026

fix(proxy): look up hashed key names with two spend log rows per key #43656

Merged

8 tasks done

yassin-berriai mentioned this pull request Oct 14, 2026

fix(caching): make Redis Cluster commands wait for a pool slot instead of tripping the circuit breaker #46488

Merged

4 of 6 tasks

kerry-berri mentioned this pull request Oct 14, 2026

fix(proxy): reserve all standalone TPM counters in one Lua call #46494

Closed

3 of 5 tasks

Conversation

devin-ai-integration Bot commented Sep 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TLDR

Files changed

Request flow

User Flow

Relevant issues

Linear ticket

Pre-Submission checklist

Screenshots / Proof of Fix

Before (e8affcb, the #43369 tip this PR is stacked on)

POST /v1/chat/completions, non-streaming, usage-based routing

POST /v1/chat/completions, non-streaming, simple-shuffle

POST /v1/chat/completions, streaming

POST /v1/responses

POST /v1/messages

POST /v1/chat/completions, non-streaming, usage-based routing

POST /v1/chat/completions, non-streaming, simple-shuffle

POST /v1/chat/completions, streaming

POST /v1/responses

POST /v1/messages

Admin UI at 6c6354a

Live re-check at ec4dd90

Type

Caveats (if any)

Medium

Low

Final Attestation

Uh oh!

devin-ai-integration Bot commented Sep 27, 2026

Uh oh!

CLAassistant commented Sep 27, 2026

Uh oh!

codecov Bot commented Sep 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

greptile-apps Bot commented Sep 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

devin-ai-integration Bot commented Sep 27, 2026

Uh oh!

Uh oh!

Uh oh!

devin-ai-integration Bot commented Sep 27, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

devin-ai-integration Bot commented Sep 27, 2026

Uh oh!

devin-ai-integration Bot commented Sep 29, 2026

Uh oh!

devin-ai-integration Bot commented Sep 29, 2026

Uh oh!

devin-ai-integration Bot commented Sep 29, 2026

Uh oh!

devin-ai-integration Bot commented Sep 29, 2026

Uh oh!

yassin-berriai commented Sep 29, 2026

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

codspeed Bot commented Sep 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Uh oh!

Uh oh!

This branch is waiting to be deployed

Reviewers

devin-ai-integration Bot commented Sep 27, 2026 •

edited

Loading

Before (`e8affcb`, the #43369 tip this PR is stacked on)

Admin UI at `6c6354a`

Live re-check at `ec4dd90`

codecov Bot commented Sep 27, 2026 •

edited

Loading

greptile-apps Bot commented Sep 27, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

codspeed Bot commented Sep 29, 2026 •

edited

Loading